Not Applicable
Not Applicable
The present invention relates generally to methods and apparatus for the determination, monitoring and prediction of various levels of alertness, and for the design and validation of fatigue countermeasures, and more particularly to methods and apparatus for the automatic characterization, detection and classification of microsleep events through processing physiological, eye tracking, video, performance and other alertness-related data obtained from a person while he or she is performing a primary task.
Impaired alertness accompanied by short microsleep events is a frequently reported phenomenon in all areas of modem life. A microsleep event can be defined as a somewhat unexpected short episode of sleep, between 1 and 30 seconds, that occurs in the midst of ongoing wakeful activity. It is suspected that such microsleep events are responsible for many accidents on the road and in the workplace, especially during nighttime. For example, the most notorious industrial accidents of our time, Three Mile Island, Bhopal, Chernobyl, and the Exxon Valdez, all occurred in the middle of the night. Microsleep events can be identified through close inspection of physiological, performance and behavioral data. The potentially serious consequences of microsleep events were demonstrated in a study on alertness levels of locomotive operators (see L. Torsvall, T. Akerstedt; xe2x80x9cSleepiness on the job: continuously measured EEG in train driversxe2x80x9d; Electroencephalography and Clinical Neurophysiology 66 (1987), pp.502-511.) During this study, one operator failed twice to respond to a stop signal, because several microsleep events occurred at the time the train passed the signal. The microsleep events were indicated clearly in the encephalogram (hereinafter referred to as EEG) and electrooculogram (hereinafter referred to as EOG) recordings.
It is well known in the art that information related to alertness, microsleep events, arousal""s, sleep stages and cognition may be discerned from changes in EEG and EOG readings. Unfortunately, not all microsleep events are as easily recognizable as the microsleep events in the aforementioned study of locomotive operators. Often, microsleep events exhibit very complex and diverse characteristics depending on the type of physiological, performance or behavioral parameter used for the detection. Furthermore, the characterization of microsleep events is strongly related to the individual person (e.g., EEG type, age, gender, chronotype, etc.) as well as the general alertness level of the person and many other circumstances (e.g., acoustic and optical stimuli, time of day, etc.)
To solve the complex and difficult task of the automatic characterization, detection and classification of microsleep events, a pattern recognition algorithm with several components is needed. These components include for example a feature extraction system, a size normalization and scaling system, a classification system and a contextual system. A neuro-fuzzy hybrid system (e.g., see C.-T. Lin, C. S. G. Lee; A neuro-fuzzy synergism to intelligent systems, Prentice-Hall, Inc. 1996) would incorporate all the components mentioned above. In addition, neuro-fuzzy hybrid systems are numerical, model-free classifiers, which are able to improve their performance through learning from errors and through their capability to generalize even if they are working in uncertain, noisy, and imprecise environments.
In recent years, a broad variety of neural networks were used successfully for the recognition of many different patterns in physiological data. Neural networks seem to be the perfect tool for the automatic recognition, classification and interpretation of various EEG patterns, such as sleep stages (e.g., see A. N. Mamelak, J. J. Quattrochi, J. A. Hobson; Automatic staging of sleep in cats using neural networks; Electroencephalography and clinical Neurophysiology 79 (1991), PP. 52-61, S. Robert, L. Tarassenko; New method of automated sleep quantification; Medical and Biological Engineering and Computing 30 (1992), pp. 509-517, J. Pardey, S. Roberts, L. Tarassenko, J. Stradling; A new approach to the analysis of human sleep/wakefulness continuum; J. Sleep Res. 5 (1996), pp. 201-210, N. Schaltenbrand, R. Lengelle, J.-P. Macer; Neural network model: Application to automatic analysis of human sleep; Computers and Biomedical Research 26 (1993), pp. 157-171, N. Schaltenbrand, R. Lengelle, M. Toussaint, R. Luthringer, G. Carelli, A. Jacqmin, E. Lainey, A. Muzet, J.-P. Macer; Sleep stage storing using neural network model: Comparison between visual and automatic analysis in normal subjects and patients; Sleep 19 (1996), pp. 26-35, and M. Groezinger, J. Roeschke, B. Kloeppel; Automatic recognition of rapid eye movement (REM) sleep by artificial neural networks; J. Sleep Res. 4 (1995), pp. 86-91), high voltage EEG spike-and-wave patterns e.g., see G. Jando, R. M. Siegel, Z. Hovath, G. Buzsaki; Pattern recognition of the electroencephalogram by artificial neural networks; Electroencephalography and clinical Neurophysiology 86 (1993), pp.100-109), seizure-related EEG pattern (e.g., see W. R. S. Weber, R. P. Lesser, R. T. Richardson, K. Wilson; An approach to seizure detection using an artificial neural network, Electroencephalography and clinical Neurophysiology 98 (1996), pp.250-272, W. Weng, K. Khorasani; An adaptive structure neural network with application to EEG automatic seizure detection; Neural Networks 9 (1996), pp. 1223-1240, and H. Qu, J. Gotman; A patient-specific algorithm for the detection of seizure onset in long-term EEG monitoring: possible use as a warning device; IEEE Transactions on Biomedical Engineering 44 (1997)), micro-arousal (A. J. Gabor, R. R. Leach, F. U. Dowla, Automatic seizure detection using a self-organizing neural network; Electroencephalography and clinical Neurophysiology 99 (1996), pp. 257-266) and for the prediction of Alzheimer disease (e.g., see W. S. Pritchard, D. W. Duke, K. L. Coburn, N. C. Moore, K. A. Tacker, M. W. Jann, R. M. Hostetler; EEG-based, neural-net predictive classification of Alzheimer""s disease versus control subjects is augmented by nonlinear EEG measures; Electroencephalography and clinical Neurophysiology 91 (1994), pp.118-130).
The concept of neural networks is very flexible and broad one. Neural networks have been applied to monitor the present somatic state of a human subject (U.S. Pat. No. 5,601,090), to predict the danger of cerebral infarction (U.S. Pat. No. 5,590,665), to detect fear (U.S. Pat. No. 5,568,126), to create a neurocognitive adaptive computer interface based on the user""s mental effort (U.S. Pat. No. 5,447,166), to obtain quantitative estimation of blood pressure attributes and similar physiological parameters (U.S. Pat. No. 5,339,818), to establish the difference between a normal and an impaired brain state (U.S. Pat. No. 5,325,862)). None of the neural network prior art discloses the characterization, detection and classification of microsleep events for achieving the goals described herein.
Parallel to the analysis of physiological data using neural networks, a variety of alertness-monitoring systems have been invented. They are based on the determination of alertness through the response to acoustic and optic stimuli (U.S. Pat. Nos. 5,95,488, 5,243,339, 5,012,226, and 4,006,539) or through the correlation between eye and head movement (U.S. Pat. Nos. 5,583,590 and 5,561,693). Recently, a sleep detection and driver alertness apparatus (U.S. Pat. No. 5,689,241) was proposed which monitors and evaluates the temperature distribution in the facial area around the nose and mouth to detect early impending sleep. None of these alertness-monitoring prior arts discloses fuzzy logic, neural networks or any combination thereof. Furthermore sophisticated methods and apparatuses are developed for tracking the eye (U.S. Pat. Nos. 5,645,550, 5,620,436), detecting the pupil of the eye (U.S. Pat. No. 5,610,673) and recognizing facial expressions of a person (U.S. Pat. No. 5,699,449). None of the prior art disclosures in the fields mentioned above are used for the determination of the alertness or fatigue level of a person.
Based on the determination and prediction of alertness and a number of physiological parameters, known in the art as xe2x80x9cfatigue countermeasures,xe2x80x9d e.g., bio-compatible schedules, sleep strategies (U.S. Pat. No. 5,433,223) and methods for modifying a person""s circadian cycle (U.S. Pat. No. 5,304,212) have been designed. None of the prior art discloses fatigue countermeasures prior art are designed and validated based on the occurrence, characterization, detection and classification of microsleep events.
In recent publications (e.g., see S. Makeig, T.-P. Jung, Tonic, Phasic, And Transient EEG Correlates Of Auditory Awareness During Drowsiness, Cognitive Brain Research 4 (1996), pp. 15-25; S. Makeig, T.-P. Jung, T. J. Sejnowski, Using Feedforward Neural Networks To Monitor Alertness From Changes In EEG Correlation And Coherence, In: D. Touretzky, M. Mozer, M. Hasselmo (Eds), Advances in Neural Information Processing Systems 8, MIT Press, Cambridge, Mass. (1996); T.-P. Jung, S. Makeig, M. Stensmo, T. J. Sejnowski; Estimation Alertness From EEG Power Spectrum; IEEE Transactions on Biomedical Engineering 44 (1997), pp. 60-69), alertness was estimated by analyzing the correlation between a subject""s EEG power spectrum and a local error rate. The error rate was generated from the response of the subject to auditory and visual stimuli. A combination of the EEG power spectrum, a feed-forward multilayer neural network, and principle component analysis was used for the data analysis. This estimation method of measuring and quantifying a subject""s level of alertness is not based on occurrence of microsleep episodes. Also, this method of measuring and quantifying a subject""s level of alertness is dependent upon the subject being able to take time out from his or her real task to perform the test activities needed to generate a local error rate. Thus, this method and other similar techniques are not suitable when the subject is performing a task requiring his or her full attention, such as driving a vehicle.
It is a special object of the present invention to provide a novel method and apparatus for evaluating and mitigating microsleep events.
It is an object of this invention to provide a method and apparatus for automatic characterization, detection and classification of microsleep events through the simultaneous processing of physiological, eye tracking, video, performance and other alertness-related data obtained from a person while he or she is performing a primary task.
It is another object of this invention to provide a method and apparatus for detecting microsleep events as a function of a subject""s physiological measurements.
It is another object of this invention to provide a method and apparatus for detecting microsleep events as a function of a subject""s behavioral measurements.
It is yet another object of this invention to provide a method and apparatus for monitoring and quantifying a subject""s alertness level without relying upon the subject""s response to test activities unrelated to the subject""s primary task.
It is a further object of this invention to provide a neuro-fuzzy system that receives a subject""s physiological and behavioral measurements and detects the occurrence of microsleep events.
It is a further object of this invention to use examples of fatigue and non-fatigue related events to teach the neuro-fuzzy system.
A method and apparatus are presented for the automatic characterization, detection and classification of microsleep events, for determining and monitoring alertness, for predicting alertness lapses, and for designing and validating fatigue countermeasures (e.g. stimulating substances, environmental stimuli, sleep strategies, bio-compatible work schedules, or any combination thereof). The method and apparatus are based on physiological data (e.g. electroencephalogram (EEG), electrooculogram (EOG), electromyogram (EMG), electrocardiogram (ECG), etc., or any combination thereof), infrared eye tracking data (e.g. point of gaze, pupil diameter, pupil fluctuations, blinking rate, blinking rate variation, etc., or any combination thereof), video data (e.g. facial expression, facial dynamics, etc., or any combination thereof), performance data (e.g. mean and variation of reaction time, mean and variation of steering wheel activity, mean and variation of lane deviation, mean and variation of heading error, etc., or any combination thereof) and other alertness-related data (e.g. electrodermal activity (EDA), blood pressure, respiration, etc., or any combination thereof) of a person while he/she continues to perform a primary task such as driving a vehicle, operating machinery, etc. The method and apparatus uses a neuro-fuzzy hybrid system for analyzing the recorded data
In accordance with one preferred embodiment of the invention, the neuro-fuzzy hybrid system consists of a combination data recording systems, feature extraction, normalization, and scaling systems, example selection systems, event classification systems, event detection systems and contextual systems. Various methods like K-means algorithm, fuzzy C-means algorithm, Principle Component Analysis (PCA), Sammon"" algorithm, Learning Vector Quantization (LVQ), Self-Organizing Map (SOM), Adaptive Resonance Theory (ART), etc., and other methods, or any combination thereof, and different neural network types such as Hopfield networks, Boltzmann machines, Multilayer Perceptron networks (MLPs), Radial-Basis Function networks (RBFs), Higher Order Neural Networks (HONNs), Probabilistic Neural Networks (PNNs), etc., and other neural networks, or any combination thereof may be incorporated in the neuro-fuzzy hybrid system.
Characteristic features of fatigue-related events which are either corresponding to microsleep events or to various transitional events (e.g. transition microsleep/non-microsleep, transition microsleep/sleep stage 1, transition microsleep sleep stage 2, etc.) like nodding off, partial and complete prolonged eyelid closure, head snapping, multiple eye blinks, blank stares, wide eyes, yawing, slow rolling eye movements, etc. are extracted for each of the physiological, eye tracking, video, performance and other alertness-related data. Feature vectors are constructed and used as input for the different algorithm and neural networks described above.
The neuro-fuzzy hybrid system works on three training levels. At the first and most tailored level, the neuro-fuzzy hybrid system is trained only with person-specific data, containing examples of microsleep events, non-microsleep events and a number of transitional events. At the second level, the neuro-fuzzy hybrid system is trained with classified data (e.g. EEG type, gender, age, chronotype, etc.). The training data set for third level contains all data subsets from levels one and two and is therefore the most common level. The final system output is a weighted combination of all three training levels, depending on how much person-specific data is available and how similar the person-specific data is to already classified data. Based on the occurrence of all detected events, alertness parameters such as mean and variability and circadian pattern of alertness, number of alertness lapses per time period, periodicity of alertness lapses, and any combinations thereof are determined and can be used for predicting alertness and designing and validating fatigue countermeasures (e.g. bio-compatible work schedules, sleep strategies, methods for modifying circadian cycle, etc.). The determination of number of alertness lapses per time period and the validation of fatigue countermeasures are demonstrated by empirical data, e.g., by means of EEG data obtained from a driver simulation study.